-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Dr.CI flaky FP when GH fails to dispatch the workflow #4998
Fix Dr.CI flaky FP when GH fails to dispatch the workflow #4998
Conversation
@huydhn is attempting to deploy a commit to the Meta Open Source Team on Vercel. A member of the Team first needs to authorize it. |
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
||
// Has log and failure lines and is a workflow job | ||
mockFailure.workflowId = "A"; | ||
expect(await isLogClassifierFailed(mockFailure)).toEqual(false); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this line supposed to be the same as the one above it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tries to add 2 tests here:
- The first one without workflow ID to mark it as a workflow run,
isLogClassifierFailed
returns false as the default value - The second one with workflow ID (so it's a job) +
hasS3Log
returning true + has failure lines, soisLogClassifierFailed
should return false
So yeah, both of them expect false as the returned value.
@@ -334,6 +334,7 @@ describe("Test various utils used by Dr.CI", () => { | |||
}); | |||
|
|||
test("test isInfraFlakyJob", () => { | |||
// Not a workflow job |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe im reading this wrong, but do you have any tests where it would have been flaky if it were a workflow job, but not if it is a workflow run?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are examples from the original PR #4622 that implements isInfraFlakyJob
. They were all jobs that weren't run. GitHub records are gone, but they are still shown on HUD https://hud.pytorch.org/pr/110608
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By querying Rockset, I can confirm that there were infra flaky jobs from 110608. Here is an example:
workflowID = 6420960240 <-- Has workflow ID, so this is a job
id = 17434762139
runnerName = '' <-- Not having a runner name is a signal for infra flaky
authorEmail = 'eltociear@gmail.com'
name = 'pull / linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge)'
jobName = 'linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge)'
conclusion = 'failure'
failure_captures = null
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry what I mean is that I don't see tests where the existence of a workflowID has an effect on the output of the function, I only see cases that returned false regardless of whether or not the workflowID is there. This makes me think that the original code would have also returned false, so I'm confused as to if the code works, am I missing something?
Fixes #4987
Dr.CI logic to detect
isInfraFlakyJob
andisLogClassifierFailed
has a FP where it misclassifies the GH failure to dispatch the whole workflow as flaky, for example pytorch/pytorch#121317.These logic should only be applicable to workflow job, not workflow run. The way to separate them is to check the
workflowId
field where it is set tonull
whenever it is a workflow run.Testing
Unit test + local curl command will mark them as legit failures: